1. Field of the Invention
The present invention relates generally to software verification and, more particularly, to the automated testing of software.
2. Related Art
Newly developed software programs must be thoroughly tested in order to eliminate as many "bugs" or errors as possible before the software is released for widespread public use. Several different methods for testing software programs have been developed. One conventional approach, generally referred to as beta testing, involves distributing the program, typically under a non-disclosure agreement, to a group of users who use the program for a period of time and report any errors which are encountered to the software developer. Although this type of testing is commonly used in the software industry, it is often found to be very time consuming, adversely affecting the scheduled release of products incorporating the software program. In addition, beta testing can be extremely difficult to control, particularly when a large number of users are provided the beta version of the software. Furthermore, due to the non-systematic use of the program, there is no guarantee that every error, or even most errors, will be identified with this approach, even under circumstances where a large number of users are using the software.
Another conventional method for verifying and testing software programs involves the use of automated software testing techniques alone or in combination with beta testing. Automatic testing generally consists of dynamically verifying that a program is consistent with a detailed specification of its behavior. Automatic test techniques generally include what is commonly referred to as basis path test and range test techniques. Basis path testing involves determining the possible paths which the software program may follow. A path is a sequential list of program state transitions. The state transitions are executed to verify specific paths through the program. Range testing involves verifying that the program can successfully perform certain functions that depend upon inputs to the program. The software is repeatedly executed while the inputs are set to one or more values within a specified range. The values can be numerical ranges or other ranges such as string lengths.
A drawback to conventional automated software testing techniques include performing path testing randomly or on only certain paths which are likely to be executed during typical program operations. Likewise, conventional automated range testing techniques often test a limited number of numerical values within and outside of the specified range of a given input. These approaches to testing software are often based upon a cost/benefit analysis: the cost associated with test and verification as compared with the probability that more extensive testing will uncover errors that would be problematic for the end user. This risk analysis results in the incomplete testing of the software program, particularly as the cost for the manual creation and subsequent maintenance of test procedures for each software program increases.
Typically, conventional automated software test techniques include the development of multiple test procedures, each directed towards the verification of a particular software program. A drawback to these test systems is that the software programs are often a collection of modules, subroutines or other components that perform smaller or simpler subtasks. Oftentimes, these smaller functional programming components are also included in, called or referenced by other, larger programs as well. As a result, procedures for verifying these common features are often found in the multiple test procedures. There are a number of drawbacks resulting from this redundancy. First, improvements made to one test procedure must often be incorporated into other test procedures as well, further increasing the cost of maintaining the automated test system. Such changes may be due to, for example, modifications resulting from prior software testing, specification changes, etc. Under such conditions, test procedures become invalid and unusable until the modifications are completed. This interdependency of test procedures overburdens the software test process, extending the time required to release a new software product.
Another drawback to conventional software test techniques is that they are directed towards the coverage of internal components or elements associated with the software, such as statements, branches or dataflow chains. Creation of such test procedures generally requires access to the source code of the software program. However, such access is not always possible because the program source code is frequently subject to copyright or trade secret protection. In addition, such software test techniques require an intimate knowledge of the software program, oftentimes requiring the software developers themselves to create the test procedures. This prevents the assignment of such tasks to less costly and more available personnel. Also, the testing of the internal components or elements may be unnecessary, and therefore wasteful. This is because there are many more possible paths to test of the internal components than necessary to insure that the functional requirements are satisfied.
What is needed, therefore, is a system and method for thoroughly and efficiently testing software programs. The system should be capable of being easily and quickly modified to accommodate changes in the software program. The system should include the ability to automatically determine the requisite tests necessary to meet user-specified criteria. Furthermore, such an automated test system should test software according to its external characteristics and functional specifications and not require access to, or an intimate knowledge of, the software program.